Spherical RoPE#2082
Conversation
set true directly when use spherical rope
|
@csjfwang Can you please post the relevant hedgedoc link here? So it is easy to find later |
Yes! Here: |
sophie-xhonneux
left a comment
sophie-xhonneux
left a comment
There was a problem hiding this comment.
Please allow for backwards compatability the rope_2D option to set rope_mode to 2d. Add a warning to the logger that this will be deprecated.
Also can you please add the performance cost in iterations/second to the hedgedoc or this PR
|
@csjfwang It would be great to get this PR over the line asap! |
Thank you @sophie-xhonneux ! |
Here are some performance cost: I've solved the above two comments, could you please help review this PR again? |
| healpix_level: 5 | ||
|
|
||
| rope_2D: False | ||
| # Generalized RoPE selector. |
There was a problem hiding this comment.
It would make more sense to have a section positional_encoding in the future
| healpix_level: 5 | ||
|
|
||
| rope_2D: False | ||
| # Generalized RoPE selector. |
There was a problem hiding this comment.
We should not add this to all configs. It should default to the case that it's not used (for the moment).
| return apply_rotary_pos_emb(q, k, cos, sin, unsqueeze_dim=unsqueeze_dim) | ||
|
|
||
|
|
||
| #################################################################################################### |
There was a problem hiding this comment.
Can we avoid these non-standard comments. Use a separate file if you think it should be separated better
There was a problem hiding this comment.
Thank you @clessig !
I've solved the above issues, could you please have a look again?
3 participants
Description
1st draft version of spherical rope.
Issue Number
Is this PR a draft? Mark it as draft.
Checklist before asking for review
./scripts/actions.sh lint./scripts/actions.sh unit-test./scripts/actions.sh integration-testlaunch-slurm.py --time 60